Systems and methods for adaptive input thresholding

ABSTRACT

The disclosed computer-implemented method may include detecting, by a computing system, a gesture that appears to be intended to trigger a response by the computing system, identifying, by the computing system, a context in which the gesture was performed, and adjusting, based at least on the context in which the gesture was performed, a threshold for determining whether to trigger the response to the gesture in a manner that causes the computing system to perform an action that is based on the detected gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate a number of exemplary embodimentsand are a part of the specification. Together with the followingdescription, these drawings demonstrate and explain various principlesof the present disclosure.

FIG. 1 is an illustration of an example block diagram of system fordetecting and responding to a gesture.

FIG. 2 is an illustration of an example gesture source included in anexample receiving device.

FIG. 3 is an illustration of a first example graph of a frequencydistribution for an input gesture versus a recognizer score.

FIG. 4 is an illustration of a second example graph of a frequencydistribution for an input gesture versus a recognizer score.

FIG. 5 is an illustration of a third example graph of a frequencydistribution for an input gesture versus a recognizer score.

FIG. 6 is an illustration of an example block diagram of a gestureresponse identifier.

FIG. 7 is a flow diagram of an exemplary computer-implemented method 700for recognizing and responding to an intended gesture.

FIG. 8 is a flow diagram of an exemplary method for adaptive inputthresholding based on the temporal costs of error correction and/or usertasks.

FIG. 9 is a block diagram of an example system that includes modules foruse in adaptive input thresholding based on the temporal costs of errorcorrection and/or user tasks.

FIG. 10 illustrates an exemplary network environment in which aspects ofthe present disclosure may be implemented.

FIG. 11 is an illustration of exemplary augmented-reality glasses thatmay be used in connection with embodiments of this disclosure.

FIG. 12 is an illustration of an exemplary virtual-reality headset thatmay be used in connection with embodiments of this disclosure.

FIG. 13 is an illustration of exemplary haptic devices that may be usedin connection with embodiments of this disclosure.

FIG. 14 is an illustration of an exemplary virtual-reality environmentaccording to embodiments of this disclosure.

FIG. 15 is an illustration of an exemplary augmented-reality environmentaccording to embodiments of this disclosure.

FIGS. 16A and 16B are illustrations of an exemplary human-machineinterface configured to be worn around a user's lower arm or wrist.

FIGS. 17A and 17B are illustrations of an exemplary schematic diagramwith internal components of a wearable system.

Throughout the drawings, identical reference characters and descriptionsindicate similar, but not necessarily identical, elements. While theexemplary embodiments described herein are susceptible to variousmodifications and alternative forms, specific embodiments have beenshown by way of example in the drawings and will be described in detailherein. However, the exemplary embodiments described herein are notintended to be limited to the particular forms disclosed. Rather, thepresent disclosure covers all modifications, equivalents, andalternatives falling within the scope of the appended claims.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

A user may interact with a computing device in a variety of ways toprovide input to, or otherwise control, actions performed by thecomputing device. For example, the user may type on a keyboard, whichmay be a mechanical keyboard interfaced with the computing device or avirtual keyboard that may be a keyboard displayed on a touchscreen of adisplay device of the computing device. The user may use a pointingdevice (e.g., a mouse) to click-on or select a button or icon displayedon the display device of the computing device. The user may interactwith buttons, icons, and/or other controls provided on a touchscreen ofthe computing device. The user may use voice controls by speaking voicecommands into a microphone of the computing device.

In addition, or in the alternative, the computing device may recognizeand interpret one or more gestures performed by the user that mayprovide control of the computing device. In some cases, the user mayperform the gesture while contacting the computing device (e.g., swipinga finger of the user across the screen of the computing device). Inother cases, the user may perform the gesture without necessarily makingany physical contact with the computing device.

In some circumstances, the user may be interacting with a computingdevice while doing other activities, while under duress, or while undercircumstances that provide less than optimal conditions for theinteraction. Such circumstances may cause the user to inaccuratelyprovide input to the computing device causing the computing device tomisinterpret the intentions of the user. In addition, or in thealternative, the circumstances may cause the computing device tomisinterpret the intentions of the user. These misinterpretations maycause the computing device to erroneously accept or reject input to thecomputing device causing the computing device to perform an action inerror and/or to not perform an action in error, respectively. In othercircumstances, the user may be interacting with a computing device andnoise or other types of interference may be introduced making itdifficult for the computing device to correctly interpret the intentionsof the user even if the user accurately provided input to the computingdevice.

In order to create an optimal user experience, a confidence thresholdfor use by the computing device for accepting or rejecting input to thecomputing device may be adaptively adjusted based on the temporal costsof correcting actions performed in error based on user inputmisinterpreted by the computing device.

The present disclosure is generally directed to adaptive inputthresholding for the recognition of a gesture as an input to a computingdevice based on the temporal costs of error correction and/or usertasks. As will be explained in greater detail below, embodiments of thepresent disclosure may provide a computing system that may detect agesture that appears to be intended to trigger a response by thecomputing system.

Once detected, the computing system may identify a context in which thegesture was performed. An example context may be a number ofapplications actively running on the computing device of the user whenthe gesture is detected. This context may indicate a degree ofmultitasking by the user when performing the gesture (e.g., the moreapplications that are actively running the greater the degree ofmultitasking). Another example context may be a current activity levelof the interaction of the user with the application when the gesture isdetected. This context may indicate a degree of time pressure that theuser was under when performing the gesture (e.g., the higher theactivity level the greater the degree of time pressure).

Based at least on the context in which the gesture was performed, athreshold for determining whether to trigger the response to the gesturemay be adjusted. Referring to the above examples, for example, thedegree of multitasking and/or the degree of time pressure may adjust athreshold for determining a recognition of the gesture and itsassociated response. Triggering the response may then cause thecomputing system to perform an action that is based on the detectedgesture.

In addition, or in the alternative, the triggering of the response maybe further based on the temporal costs of performing the action in error(a false positive) and/or the temporal costs of not performing theaction in error (a false negative). Therefore, the computing system maydetermine a confidence level or score in association with the detectedgesture based on the context in which the gesture was performed and thetemporal costs of error correction of performing or not performing anaction based on the detected error.

Features from any of the embodiments described herein may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

The following will provide, with reference to FIGS. 1 and 6, anexemplary system for recognizing and responding to gestures, withreference to FIGS. 3-5, exemplary graphs of a frequency distribution foran input gesture versus a recognizer score, and with reference to FIGS.7 and 8, exemplary methods for recognizing and responding to an intendedgesture using adaptive input thresholding based on the temporal costs oferror correction and/or user tasks. FIGS. 2 and 12-17 provide examplesof systems, devices, and environments that may incorporate the use ofthe exemplary system for recognizing and responding to gestures.

FIG. 1 is an illustration of an example block diagram of a system 100for detecting and responding to a gesture. The system 100 may include agesture source 102 and a receiving device 104. In some implementations,the gesture source 102 and the receiving device 104 may be differentdevices. In some cases, the gesture source 102 may be physicallyconnected to the receiving device 104 for communication purposes. Insome cases, the gesture source 102 may be wirelessly connected to thereceiving device 104. In some implementations, the receiving device 104may include the gesture source 102. For example, the gesture source 102may be included within or be physically a part of the receiving device104.

Gestures as referred to herein may be motion gestures, which aregestures made by a user of a gesture source without the user making anydirect physical contact with a receiving device when performing thegesture. Gestures as referred to herein may also be contact gestures,which are gestures made by a user of a gesture source where the gesturesource may be included in the receiving device and the user performs thegesture while making contact with the gesture source included in thereceiving device.

A gesture source may interface with a receiving device in a multitude ofways. For example, the gesture source 102 may interface with thereceiving device 104. In some implementations, the gesture source 102may interface with a gesture receiver 106 included in the receivingdevice 104. The gesture receiver 106 may communicate with the gesturesource 102 using one or more communication protocols. In someimplementations, the gesture source 102 may communicate and/or interfacewith the receiving device 104 using a wireless communication protocolsuch as WiFi or BLUETOOTH as described herein with reference to FIG. 10.In some implementations, the gesture source 102 may communicate and/orinterface with the receiving device 104 by way of one or more wiredcommunication protocols such as Universal Serial Bus (USB) as describedherein with reference to FIG. 10.

A gesture receiver may receive information and data from a gesturesource as it relates to a gesture as performed by a user of the gesturesource. In some implementations, the gesture source 102 and thereceiving device 104 may be different computing devices. In an exampleimplementation, the receiving device 104 may be a wearable mobilecomputing device (e.g., augmented-reality glasses as shown in FIG. 11, ahead-worn display system such as virtual-reality system 1200 as shown inFIG. 12). The gesture source 102 may be another handheld or mobilecomputing device. In some implementations, the gesture source 102 mayalso be a wearable mobile computing device (e.g., a wearable glove(haptic device 1310) and wristband (haptic device 1320) as shown in FIG.13, a wristband (haptic device 1530) as shown in FIG. 15, a wearablesystem 1600 as shown in FIGS. 16A-B and 17A-B). Such gestures may beconsidered motion gestures.

FIG. 2 is an illustration of an example gesture source included in anexample receiving device. In some implementations, referring to FIG. 1,the gesture source 102 may be part of or included in the receivingdevice 104. In an example implementation, a receiving device (e.g., thereceiving device 104) may be a handheld or mobile computing device(e.g., computing device 202). A gesture source (e.g., the gesture source102) may be a touchscreen of a computing device (e.g., touchscreen 204of the computing device 202). In this example implementation, thegesture receiver 106 may receive information and data related to userinteractions with the touchscreen while running an application on thecomputing device. The user interactions may be gestures made by the userwhile contacting the touchscreen 204 of the computing device 202 (e.g.,contact gestures).

A gesture receiver may include hardware and/or software forcommunicating with, interfacing with, and/or monitoring the gesturesource to determine gestures as performed by a user when interactingwith the gesture source. Referring to FIG. 1, for example, the gesturereceiver 106 may receive information and data related to gestures asperformed by a user interacting with the gesture source 102. The gesturereceiver 106 may provide the gesture related information and data to agesture detector 108. The gesture detector 108 may determine if thereceiving device 104 should interpret the gesture related informationand data as an input gesture for the system 100.

FIG. 3 is an illustration of a first example graph 300 of a frequencydistribution for an input gesture (frequency distribution 302) versus arecognizer score 304. For a curve 306, the frequency distribution 302may represent a frequency of an intention of a user to have a gesture ofthe user that is recognized and assigned an associated recognizer scorenot initiate an action on the computing device of the user because theuser did not intend to perform the gesture.

FIG. 4 is an illustration of a second example graph 400 of a frequencydistribution for an input gesture (the frequency distribution 302)versus the recognizer score 304. For a curve 406, the frequencydistribution 302 may represent a frequency of an intention of a user tohave a gesture of the user that is recognized and assigned an associatedrecognizer score initiate an action on the computing device of the userbecause the user intended to perform the gesture.

FIG. 5 is an illustration of a third example graph 500 of a frequencydistribution for an input gesture (the frequency distribution 302)versus the recognizer score 304. The graph 500 combines the first graph300 and the second graph 400. The frequency values for the gesture tonot initiate an action on the computing device as shown in the curve 306and the frequency values for the gesture to initiate an action on thecomputing device as shown in the curve 406 may be normalized using thesame normalization factor. As such, the curve 306 and the curve 406 maybe plotted using the same frequency distribution axis (the frequencydistribution 302) and the same recognizer score axis (the recognizerscore 304).

In some implementations, a system (e.g., the system 100 as shown inFIG. 1) may gather recognizer scores output by a gesture detector (e.g.,the gesture detector 108) that represent a confidence in the accuracy ofthe detection or recognition of a gesture. The system may continuouslyrecord the recognizer scores while a user performs tasks on a computingdevice of the user. The tasks may include intentional performance of oneor more gestures. In addition, or in the alternative, the tasks mayinclude the unintentional performance of one or more gestures. Inaddition, or in the alternative, the tasks may include the inaccuratedetection of the performance of one or more gestures.

The system may label the gathered data accordingly to indicate when auser was intentionally performing a recognized gesture, and to indicatewhen a user was unintentionally performing a recognized gesture. Thesystem may create two distributions. A first distribution may includedata for gesture recognition recognizer scores associated with the userintending to perform the gesture. The system may use the data for thefirst distribution to generate the curve 406. A second distribution mayinclude data for gesture recognition recognizer scores associated withthe user not intending to perform the gesture (e.g., the user wasperforming other activities). The system may use the data for the seconddistribution to generate the curve 306. The curve 406 and the curve 306,therefore, may represent the relative frequency of an intention of auser have a detected gesture of a particular recognizer score initiatean action on the computing device versus the relative frequency of adetected gesture of a particular recognizer score not to initiate anaction on the computing device, respectively.

A system may generate a gesture response model that maps gesturerecognition recognizer scores to a relative frequency of an intention ofa user to have a recognized gesture initiate an action on a computingdevice of the user and to a relative frequency of an intention of a userto have a recognized gesture not initiate an action on a computingdevice of the user. For example, referring to FIGS. 1, 3, 4 and 5, asystem 100 may use the curve 306 and the curve 406 to generate a gestureresponse model that maps recognizer scores output by the gesturedetector 108 to an intention of a user to either to perform the gesture(curve 406) or not perform the gesture (curve 306). The system 100 mayuse the gesture response model to determine a probability of anintention of the user given a recognizer score for a gesture. Forexample, the system 100 may determine that for a recognizer score of 0.6there may be a 33% chance that the user is trying to perform thegesture, and a 66% chance that the user is not trying to perform thegesture.

Referring to FIGS. 1 and 3, the gesture response identifier 110 maycalculate a normalized set of frequencies versus a recognizer score fora detected gesture as shown, for example, by the curve 306. The curve306 may represent the relative frequency of a detected gesture of aparticular recognizer score not to initiate an action on the computingdevice. The gesture response identifier 110 may calculate a gesturerecognition recognizer score that indicates a clarity of the gesture asdetected by the gesture detector 108 based on a probability that thesystem 100, and specifically the gesture detector 108, has accuratelydetected the gesture. In some implementations, a context in which a userperforms the gesture may impact the clarity of the information and datafor the gesture as provided by the gesture source 102 to the gesturereceiver 106.

Referring to FIG. 1, the system 100, and specifically the gestureresponse identifier 110, may identify input gesture of a particularnormalized recognizer score that may fall within or under the curve 306for gestures identified as not intended to trigger a response by thesystem 100. Such gestures may not trigger the receiving device 104 torespond to the gestures. As such, the gesture response identifier 110may not provide the detected gesture to an action executer 112 forperforming an action on the receiving device 104 based on the detectedaction.

In some cases, referring to FIG. 2, a user interacting with atouchscreen of a computing device (e.g., the touchscreen 204 of thecomputing device 202) may not fully complete a gesture. For example, theuser may start but not complete a swipe because the user decided not tocomplete a particular transaction in an application running on thecomputing device 202. In some cases, a user interacting with a gesturesource that is wirelessly interfaced to the receiving device may notfully perform or complete a gesture. For example, a handheld controllermay interface wirelessly to an exemplary virtual-reality headset (e.g.,virtual-reality system 1200 as shown in FIG. 12). While starting toperform a gesture using the handheld controller as the gesture source,the user may be interrupted and may then put down the controller, notcompleting an intended gesture for input to the virtual-reality headset.

Referring to FIGS. 1 and 4, the gesture response identifier 110 maycalculate a normalized set of frequencies versus a recognizer score fora detected gesture as shown, for example, by the curve 406. The curve406 may represent the relative frequency of a detected gesture of aparticular recognizer score to initiate an action on the computingdevice. The gesture response identifier 110 may calculate a gesturerecognition recognizer score that indicates a clarity of the gesture asdetected by the gesture detector 108 based on a probability that thesystem 100, and specifically the gesture detector 108, has accuratelydetected the gesture. In some implementations, a context in which a userperforms the gesture may impact the clarity of the information and datafor the gesture as provided by the gesture source 102 to the gesturereceiver 106.

Referring to FIG. 1, the system 100 and specifically the gestureresponse identifier 110 may identify input gesture of a particularnormalized recognizer score that may fall within or under the curve 406for gestures identified as intended to trigger a response by the system100. Such gestures may trigger the receiving device 104 to respond tothe gesture. As such, the gesture response identifier 110 may providethe detected gesture to an action executer 112 for performing an actionon the receiving device 104 based on the detected action.

In some cases, referring to FIG. 2, a user interacting with atouchscreen of a computing device (e.g., the touchscreen 204 of thecomputing device 202) may fully complete a gesture. For example, theuser may complete a swipe in an ecommerce application running on thecomputing device 202 to initiate a purchase transaction in theapplication. In some cases, a user interacting with a gesture sourcethat is wirelessly interfaced to the receiving device fully perform andcomplete a gesture. For example, referring to FIGS. 16A-B and 17A-B, awearable system may interface wirelessly to head-mounted display system.Examples of a head-mounted display system may include, but are notlimited to, a virtual-reality headset (e.g., virtual-reality system 1200as shown in FIG. 12) and augmented-reality glasses (e.g.,augmented-reality glasses as shown in FIG. 11). The user may perform agesture with the wearable system to initiate an action in an applicationrunning in the head-mounted display system.

Referring to FIG. 5, the third example graph 500 may represent a modelof how recognizer scores output by a gesture detector map to a trueintention of a user to either perform a gesture (curve 406) or notperform a gesture (curve 306). However, as shown by the graph 500, anoverlapping area 502 may identify normalized frequencies for gestureinformation and data of gesture recognition recognizer scores that couldbe interpreted as gestures that are not intended to trigger a responseby the system 100 or as gestures that are intended to trigger a responseby the system 100.

Referring to FIGS. 1 and 5, for example, the gesture response identifier110 may identify input gestures of recognizer scores at or below (lessthan) a lower threshold recognizer score 506 (e.g., at or below arecognizer score of approximately 0.57) as gestures that are notintended to trigger a response by the system 100. In another example,the gesture response identifier 110 may identify input gestures ofrecognizer scores at or above (greater than) an upper thresholdrecognizer score 508 (e.g., at or above a recognizer score ofapproximately 0.72) as gestures that are intended to trigger a responseby the system 100. In another example, the system 100 may identify inputgestures of a recognizer score of 0.6 having a 33% chance that the useris trying to perform the gesture, and a 66% chance that the user is nottrying to perform the gesture.

A gesture response identifier may determine and set a gesturerecognition threshold confidence level. The gesture response identifiermay use the gesture recognition threshold confidence level to determinewhether a gesture received and detected by a computing device wasintended to trigger a response or action on the computing device. Forexample, referring to FIG. 1, the gesture response identifier 110 maydetermine and set a gesture recognition threshold confidence level 504.The gesture response identifier 110 may use the gesture recognitionthreshold confidence level 504 to identify input gestures of recognizerscores at or below (less than) the gesture recognition thresholdconfidence level 504 as gestures that are not intended to trigger aresponse by the system 100. The gesture response identifier 110 may usethe gesture recognition threshold confidence level 504 to identify inputgestures of recognizer scores at or above (greater than) the gesturerecognition threshold confidence level 504 as gestures that are intendedto trigger a response by the system 100.

A gesture response identifier may adjust the gesture recognitionthreshold confidence level based on one or more criteria. For example,the gesture response identifier 110 may adjust the gesture recognitionthreshold confidence level 504 based on a context in which a userperformed the gesture. In some situations when interacting with thegesture source 102, a context in which the user performed the gesturemay include, but is not limited to, a user being under time pressure, auser being distracted, a user being interrupted, a user changing theirmind before completing the gesture, and/or a user multitasking whileinteracting with the gesture source (e.g., interacting with multipleapplications running on the computing device of the user, walking whileperforming the gesture, etc.). In other situations when interacting withthe gesture source 102, a context in which the user performed thegesture may include, but is not limited to, a time of day, a level ofambient lighting, a state of the computing device of the user, and/or alocation where the gesture was performed.

In another example, the gesture response identifier 110 may adjust thegesture recognition threshold confidence level 504 based on a temporalcost of error correction for an action that may have been erroneousperformed (or not performed) based on erroneous recognition (or ignoringof) a detected gesture. For example, the gesture response identifier 110may adjust the gesture recognition threshold confidence level 504 basedon a time to correct a false positive error (erroneously performing anunintended action). In another example, the gesture response identifier110 may adjust the gesture recognition threshold confidence level 504based on a time to correct a false negative error (erroneously notperforming an intended action).

In some implementations, a gesture response identifier may adjust thegesture recognition threshold confidence level based on one or morecriteria associated with a specific detected gesture. For example,referring to FIG. 1, the gesture response identifier 110 may dynamicallyadjust the gesture recognition threshold confidence level 504 for eachdetected gesture based on criteria associated with the detected gesturesuch as a context in which the user performed the gesture. Dynamicallyadjusting the gesture recognition threshold confidence level 504 mayprovide a context dependent adjusted threshold for determining if adetected gesture is intended to trigger a response by the system 100.Specifically, dynamically adjusting the gesture recognition thresholdconfidence level 504 for each detected gesture provides an improvedsystem for gesture recognition for detected gestures whose normalizedfrequencies are of a recognizer score that may be below (less than) theupper threshold recognizer score 508 yet above (greater than) the lowerthreshold recognizer score 506.

FIG. 6 is an illustration of an example block diagram of a gestureresponse identifier (e.g., the gesture response identifier 110 as shownin FIG. 1). The gesture response identifier 110 may include an adaptivemodulator 602 that receives criteria for use in adjusting a confidencelevel for use by a gesture recognizer 604. The gesture recognizer 604may use an adapted modulated gesture recognition threshold level (e.g.,the gesture recognition threshold confidence level 504) to determinewhether to trigger a response to a detected gesture. When the gesturerecognizer 604 determines that the response may be triggered, thetriggering of the response by the gesture recognizer 604 may cause theaction executer 112 to perform an action based on the detected gesture.

An estimator included in an adaptive modulator may use criteria for acontext in which a user performs a detected gesture to provide a gestureresponse model with an estimation of an influence or impact of thecontext on a recognizer score for the detected gesture. For example,referring to FIG. 1, an estimator 606 may receive criteria 608a-fassociated with a context of a user when interacting with the gesturesource 102.

Criterion 608 a may be a time to correct a false negative error. Forexample, the criterion 608 a may be a time-based value (e.g., seconds,minutes, hours, etc.) for how long it may take to perform a task thatthe system 100 erroneously did not perform because of erroneouslydetecting an intended gesture as unintended.

Criterion 608 b may be a time to correct a false positive error. Forexample, the criterion 608 b may be a time-based value (e.g., seconds,minutes, hours, etc.) for how long it may take to undo a task that thesystem 100 erroneously performed because of erroneously detecting anunintended gesture as intended.

Criterion 608 c may be a probability of providing an input to the systemversus not providing the input. For example, criterion 608 c may be avalue (e.g., a percentage value, a normalized value) for a probabilityof an impact to the system 100 of performing versus not performing anaction in the system 100 that is associated with the input.

Criterion 608 d may be one or more parameters associated with a currenttask of a user while interacting with a gesture source. For example, theone or more parameters may include, but are not limited to, a time ofday, a level of ambient lighting, a state of a computing device of theuser that may include the gesture source, and/or a location where theuser performed the gesture.

Criterion 608 e may be a degree of multitasking of a user whileinteracting or not interacting with a gesture source. In some cases, thedegree of multitasking may be based, at least in part, on a number ofapplications that are active (e.g., running or executing) on thecomputing device of the user that may include the gesture source. Insome cases, the degree of multitasking may be based, at least in part,on other activities the user may be performing while interacting withthe gesture source, such as walking or talking, that may cause the userto be distracted possibly leading to erroneous input gestures. In somecases, the degree of multitasking may be based, at least in part, onother activities the user may be performing while not interacting withthe gesture source that may be interpreted as input by the gesturesource, such as finger pinching actions that a user may perform whilefolding laundry, turning pages of a book, etc. that may be interpretedas input by a wrist-worn computing device of a user.

Criterion 608 f may be a degree of time pressure a user may face wheninteracting with a gesture source. For example, the degree of timepressure may be based on a number of actively running applications on acomputing device of the user that may include the gesture source, and/ora current activity of the user while inputting the gesture (e.g., theuser needs to provide the input quickly, the user is distracted by otheractivities being performed by other applications running on thecomputing device that includes the gesture source).

An estimator may generate an estimate of a combination of one or more ofthe criteria for a context in which a user performs a detected gesture.In some implementations, the estimator may use all of the criteria forgenerating the estimate. In some implementations, the estimator may usea subset of the criteria (e.g., less than all of the criteria) forgenerating the estimate. In some implementations, the estimator mayprovide a weight to one or more of the criteria when generating theestimate. For example, each weight may be based on an importance of therespective criterion in the context.

An estimator may provide an estimate of a combination of one or more ofthe criteria for a context in which a user performs a detected gestureto a gesture response model. The gesture response model may use theestimate for determining or calculating an adapted modulated gesturerecognition threshold for the detected gesture. For example, theestimator 606 may provide an estimate of a combination of one or more ofthe criteria 608 a-f to a gesture response model 610. The gestureresponse model 610 may also receive a recognizer score 612. Therecognizer score 612 may be a score for recognizing a detected gestureindependent of a context in which a user performed the detected gesture.For example, the recognizer score 612 may be based on machine learningof a confidence associated with a measure of the detected gesture.

The gesture response model 610 may use the estimate of the combinationof the one or more of the criteria 608 a-f provided by the estimator 606to further refine, adapt, or update the recognizer score 612 to generatean adapted modulated gesture recognition threshold (e.g., the gesturerecognition threshold confidence level 504) for use by the gesturerecognizer 604 when determining if a detected gesture provided by thegesture detector 108 is intended or unintended. In some implementations,the gesture response model 610 may apply a weight or importance to thecombination of the one or more of the criteria 608 a-f provided by theestimator 606 and/or to the recognizer score 612 when generating thegesture recognition threshold confidence level 504.

Referring to FIGS. 3-5, the gesture recognizer 604 may use the gesturerecognition threshold confidence level 504 to determine if a recognizedgesture is intended or unintended. If the gesture recognizer 604recognizes the gesture as intended, the gesture recognizer 604 maytrigger the response to the gesture causing the action executer 112 toperform an action based on the intended detected gesture. If the gesturerecognizer 604 recognizes the gesture as unintended, the gesturerecognizer 604 may not trigger the response to the gesture.

FIG. 7 is a flow diagram of an exemplary computer-implemented method 700for recognizing and responding to an intended gesture. The steps shownin FIG. 7 may be performed by any suitable computer-executable codeand/or computing system, including the system(s) illustrated in FIGS. 1,6, 9 and 10. In one example, each of the steps shown in FIG. 7 mayrepresent an algorithm whose structure includes and/or is represented bymultiple sub-steps, examples of which will be provided in greater detailbelow.

As illustrated in FIG. 7, at step 710 one or more of the systemsdescribed herein may generate gesture data. For example, referring toFIG. 1, the gesture source 102 may generate gesture data.

In some embodiments, the term “gesture” may refer to movement performedby a user (e.g., by a part of a body of a user, a hand, a head, etc.)for use as intended input to a computing device for controlling theoperation of the computing device. In some cases, a user may perform thegesture while contacting the computing device (e.g., swiping a finger ofthe user across the screen of the computing device). Such gestures maybe referred to as contact gestures. In other cases, the user may performthe gesture without necessarily making any physical contact with thecomputing device (e.g., the computing device may be a wearable mobilecomputing device and the gesture source may be another handheld ormobile computing device). Such gestures may be referred to as motiongestures. As discussed herein, the term “gesture” may refer to contactgestures and motion gestures.

The systems described herein may perform step 710 in a variety of ways.In one example, referring to FIG. 1, the gesture source may generategesture information and data based on a user interacting with thegesture source 102.

As illustrated in FIG. 7, at step 712 one or more of the systemsdescribed herein may receive gesture data. For example, referring toFIG. 1, the gesture receiver may receive gesture data from the gesturesource 102.

The systems described herein may perform step 712 in a variety of ways.In one example, the receiving device 104, and specifically the gesturereceiver 106, may receive information and data representative of aninteraction of a user with the gesture source 102 from the gesturesource 102. In some implementations, the gesture source 102 and thereceiving device may be different computing devices. For example, thegesture source 102 may provide information and data representative of aninteraction of a user with the gesture source 102 to the receivingdevice 104 by way of a network (e.g., a wireless network).

As illustrated in FIG. 7, at step 714 one or more of the systemsdescribed herein may determine if a gesture that appears to be intendedto trigger a response is detected (if a potential gesture is detected).For example, referring to FIG. 1, the gesture receiver 106 may providegesture related information and data to the gesture detector 108. Thegesture detector 108 may determine if the receiving device 104 shouldinterpret the gesture related information and data as a potential inputgesture for the system 100.

The systems described herein may perform step 714 in a variety of ways.In one example, the gesture detector 108 may determine that the gesturerelated information and data received by the gesture receiver 106 is fora gesture that appears to be intended to trigger a response by thereceiving device 104. If the gesture detector 108 determines that thegesture related information and data received by the gesture receiver106 is for a gesture that appears to be intended to trigger a responseby the receiving device 104 (a potential intended gesture), the methodcontinues to step 718. If the gesture detector 108 determines that thegesture related information and data received by the gesture receiver106 is an unintended gesture, the method continues to step 712.

As illustrated in FIG. 7, at step 718 one or more of the systemsdescribed herein may identify one or more parameters associated with acurrent task of a user while interacting with a gesture source. Forexample, the one or more parameters may include, but are not limited to,a time of day, a level of ambient lighting, a state of a computingdevice of the user that may include the gesture source, and/or alocation where the user performed the gesture. For example, referring toFIG. 1, in implementations where the gesture source 102 is part of(included in) the receiving device 104, the receiving device 104 mayidentify the one or more parameters using hardware and/or software onthe receiving device 104. For example, referring to FIG. 1, inimplementations where the gesture source 102 is not part of (is separatefrom) the receiving device 104, the gesture source 102 may identify theone or more parameters using hardware and/or software on the gesturesource 102. The gesture source 102 may provide the one or moreparameters to the receiving device 104.

The systems described herein may perform step 718 in a variety of ways.In one example, a clock application may provide time of day data. Inanother example, a Global Positioning System (GPS) may provide locationinformation and data. In some implementations where the receiving device104 includes the gesture source 102, the receiving device 104 may alsoinclude the clock application, the GPS, and any other hardware and/orsoftware for use in identifying the one or more parameters associatedwith a current task of the user while interacting with the gesturesource 102. In some implementations where the receiving device 104 doesnot include the gesture source 102, the gesture source 102 may includethe clock application, the GPS, and any other hardware and/or softwarefor use in identifying the one or more parameters associated with acurrent task of the user while interacting with the gesture source 102.

As illustrated in FIG. 7, at step 720 one or more of the systemsdescribed herein may identify a degree of multitasking of a user whileinteracting with a gesture source. For example, referring to FIG. 1, thesystem 100 may identify a degree of multitasking of a user whileinteracting with the gesture source 102.

In some embodiments, the term “multitasking” may refer to a userperforming more than one task while interacting with a gesture source.For example, a user may be interfacing with a touchscreen of a computingdevice while walking. In this example, the multitasking of the user, andspecifically a degree of the multitasking, may contribute to erroneousgesture detection (e.g., false negatives and/or false positives).

In some embodiments, the term “multitasking” may refer to a computingdevice executing more than one (e.g., two or more) applications on thecomputing device simultaneously. In this example, the multitasking ofthe computing device that includes the gesture source, and specificallya degree of the multitasking, may contribute to erroneous gesturedetection (e.g., false negatives and/or false positives).

In some embodiments, the term “multitasking” may refer to a userperforming multiple activities that may or may not involve a computingdevice of a user. For example, the system 100 may detect an inputgesture that was not intended to initiate an action on the system 100.For example, a user may be wearing a computing device on a wrist of theuser. The wrist-worn computing device may detect a pinch gesture.However, the user may be performing the pinch gesture while engaged inan activity independent of the wrist-worn device. Because the fingers ofthe user while engaged in this activity may be performing a pinchgesture similar to the pinch gesture for use as input to the wrist-worncomputing device, the system 100 may misinterpret the pinch gesture asintending to initiate an action on the wrist-worn computing device. Thiswould in the occurrence of a false positive error.

The systems described herein may perform step 720 in a variety of ways.In one example, a computing device that includes the gesture source 102may determine multitasking of the user and/or the computing device foruse by the system 100 in determining a degree of multitasking of theuser while performing the gesture.

As illustrated in FIG. 7, at step 722 one or more of the systemsdescribed herein may identify a degree of time pressure that the usermay be under while performing the input gesture. For example, referringto FIG. 1, the system 100 may identify a degree of time pressure thatthe user may be under while interacting with the gesture source 102.

The systems described herein may perform step 722 in a variety of ways.In one example, a computing device that includes the gesture source 102may determine a number of actively running applications on the computingdevice while the user is interacting with the gesture source that maycontribute to or be a basis for the time pressure. In another example, acomputing device that includes the gesture source 102 may determine acurrent activity of the user while interacting with the gesture sourcethat may place a time constraint on the interaction.

As illustrated in FIG. 7, at step 724 one or more of the systemsdescribed herein may determine a time (e.g., estimate a time) to correcta false negative error. For example, referring to FIG. 1, the gestureresponse identifier 110 may determine (e.g., calculate an estimation of)a time to correct a false negative error.

In some embodiments, the term “false negative” may refer to incorrectlydetermining that a gesture is unintended. Because of this determination,a gesture that was intended to trigger a response, did not trigger theresponse and subsequently an action was not performed on a computingdevice.

The systems described herein may perform step 724 in a variety of ways.In one example, the gesture response identifier 110 may determine anestimated time involved in recovering from not performing an action onthe receiving device 104 as a time to correct a false negative error. Insome implementations, a context in which the user performed the gestureusing the gesture source 102 may include an estimated time involved inrecovering from not performing the action.

As illustrated in FIG. 7, at step 726 one or more of the systemsdescribed herein may determine a time (e.g., estimate a time) to correcta false positive error. For example, referring to FIG. 1, the gestureresponse identifier 110 may determine (e.g., calculate an estimation of)a time to correct a false positive error.

In some embodiments, the term “false positive” may refer to incorrectlydetermining that a gesture is intended. Because of this determination, agesture triggered a response and subsequently an action was performed ona computing device that should not have been performed.

The systems described herein may perform step 726 in a variety of ways.In one example, the gesture response identifier 110 may determine anestimated time involved in recovering from performing an action on thereceiving device 104 that should not have been performed as a time tocorrect a false positive error. The estimated recovery time may includethe time needed to undo the action on the receiving device 104. In someimplementations, a context in which the user performed the gesture usingthe gesture source 102 may include an estimated time involved inrecovering from performing the action.

As illustrated in FIG. 7, at step 728 one or more of the systemsdescribed herein may determine a probability of a gesture as an input toa computing device. For example, referring to FIG. 1, the system 100 maydetermine a probability of a gesture as an input to the receiving device104.

In some embodiments, the term “probability” may refer to a likelihood ofan occurrence of an event. For example, a probability of a gesture beingan input may be a likelihood of a detected gesture being a particularinput to a computing device.

The systems described herein may perform step 728 in a variety of ways.In one example, the gesture response identifier 110 may determine aprobability that a detected gesture is an input to the computing device.

As illustrated in FIG. 7, at step 730 one or more of the systemsdescribed herein may determine a probability of a gesture as not beingan input to a computing device. For example, referring to FIG. 1, thesystem 100 may determine a probability of a gesture as not being aninput to the receiving device 104. For example, a probability of agesture not being an input may be a likelihood of a detected gesture notbeing a particular input to a computing device.

The systems described herein may perform step 730 in a variety of ways.In one example, the gesture response identifier 110 may determine aprobability that a detected gesture is not an input to the computingdevice.

As illustrated in FIG. 7, at step 732 one or more of the systemsdescribed herein may generate an estimate based on a context associatedwith the gesture. For example, referring to FIG. 6, the estimator 606may generate an estimate based on a combination of one or more of thecriteria for a context in which a user performs a detected gesture. Theone or more criteria are determined in steps 718, 720, 722, 724, 726,728, and 730. In some implementations, the estimator 606 may generate anestimate based on a single criteria. In some implementations, theestimator 606 may generate an estimate based on a combination of one ormore criteria that may be two or more criteria but less than allcriteria. In some implementations, the estimator 606 may generate anestimate based on all criteria.

In some embodiments, the term “context” may refer to one or more of acombination of circumstances, situations, and/or environments that forma setting for the understanding, interpreting, and/or recognition of agesture.

The systems described herein may perform step 732 in a variety of ways.In one example, the estimator 606 may generate an estimate of acombination of one or more of the criteria as determined in steps 718,720, 722, 724, 726, 728, and 730 for a context in which a user performeda detected gesture. In some implementations, the estimator 606 may useall of the criteria as determined in steps 718, 720, 722, 724, 726, 728,and 730 for generating the estimate. In some implementations, theestimator 606 may use a subset of the criteria (e.g., less than all ofthe criteria) for generating the estimate. In some implementations, theestimator 606 may provide a weight to one or more of the criteria whengenerating the estimate. For example, each weight may be based on animportance of the respective criterion in the context.

As illustrated in FIG. 7, at step 734 one or more of the systemsdescribed herein may determine a recognizer score for the gesture. Forexample, referring to FIG. 1, the system 100, and specifically thegesture response identifier 110, may determine a recognizer score forthe gesture.

In some embodiments, the term “recognizer score” may refer to a numberor value that expresses a confidence in a detected gesture as beingrecognized or intended. The gesture recognizer 604 may use the gesturerecognition threshold confidence level 504 to determine if a recognizedgesture is intended or unintended. As shown for example in FIG. 5, thegesture recognition threshold confidence level 504 may be based on therecognizer score 304 for a detected gesture.

The systems described herein may perform step 734 in a variety of ways.In one example, the recognizer score 612 may be a score for recognizinga detected gesture that may be based on machine learning of a confidenceassociated with a measure of the detected gesture.

As illustrated in FIG. 7, at step 736 one or more of the systemsdescribed herein may determine a gesture recognition thresholdconfidence level. For example, referring to FIG. 6, the gesture responsemodel 610 may generate the gesture recognition threshold confidencelevel 504.

As illustrated in FIG. 7, at step 736 one or more of the systemsdescribed herein may use an estimate for determining or generating thegesture recognition threshold confidence level 504 for the detectedgesture. For example, the estimator 606 may provide an estimate of acombination of one or more of the criteria 608a-f to the gestureresponse model 610. The gesture response model 610 may also receive therecognizer score 612. The gesture response model 610 may use theestimate of the combination of the one or more of the criteria 608 a-fprovided by the estimator 606 to further refine, adapt, or update therecognizer score 612 to generate the gesture recognition thresholdconfidence level 504.

As illustrated in FIG. 7, at step 738 one or more of the systemsdescribed herein may determine whether the system should response to thegesture. For example, referring to FIG. 6, the gesture recognizer 604may determine whether to respond to a detected gesture.

The systems described herein may perform step 738 in a variety of ways.In one example, the gesture recognizer 604 may use the gesturerecognition threshold confidence level 504 when determining if adetected gesture provided by the gesture detector 108 is intended orunintended. If the gesture recognizer 604 determines that the gesture isan unintended gesture, the method continues to step 712. If the gesturerecognizer 604 determines that the gesture is an intended gesture, themethod continues to step 740.

As illustrated in FIG. 7, at step 740 one or more of the systemsdescribed herein may perform one or more actions based on the gesture.For example, referring to FIG. 6, the gesture recognizer may determinewhether to respond to a detected gesture.

In some embodiments, the term “action” may refer to performing a processor activity on a computing device in response to and based on anintended gesture of a user. For example, an action may be executing anapplication on the computing device.

The systems described herein may perform step 740 in a variety of ways.In one example, when the gesture recognizer 604 determines that theresponse may be triggered, the triggering of the response by the gesturerecognizer 604 may cause the action executer 112 to perform an actionbased on the detected gesture.

FIG. 8 is a flow diagram of an exemplary computer-implemented method 800for adaptive input thresholding based on the temporal costs of errorcorrection and/or user tasks. The steps shown in FIG. 8 may be performedby any suitable computer-executable code and/or computing system,including the system(s) illustrated in FIGS. 1, 6, 9, and 10. In oneexample, each of the steps shown in FIG. 8 may represent an algorithmwhose structure includes and/or is represented by multiple sub-steps,examples of which will be provided in greater detail below.

As illustrated in FIG. 8, at step 810 one or more of the systemsdescribed herein may detect a gesture that appears to be intended totrigger a response by the computing system. For example, referring toFIG. 1, the system 100 may detect a gesture that appears to be intendedto trigger a response by the system 100.

In some embodiments, the term “trigger” may refer to something that maycause or initiate a particular response in a system. The response maythen be used to initiate the performance of an action by the system.

The systems described herein may perform step 810 in a variety of ways.In one example, the gesture receiver 106 may detect a gesture thatappears to be intended to trigger a response by the receiving device 104based on information and data received from the gesture source 102.

As illustrated in FIG. 8, at step 820 one or more of the systemsdescribed herein may identify a context in which the gesture wasperformed. For example, referring to FIGS. 1 and 6, an estimator 606 mayreceive criteria 608 a-f associated with a context of a user wheninteracting with the gesture source 102.

The systems described herein may perform step 820 in a variety of ways.In one example, referring to FIGS. 1, 6, and 7, the gesture source 102and/or the receiving device 104 may identify a context in which thegesture was performed as described, for example, with reference to steps718, 720, 722, 724, 726, 728, and 730.

As illustrated in FIG. 8, at step 830 one or more of the systemsdescribed herein may adjust, based at least on the context in which thegesture was performed, a threshold for determining whether to triggerthe response to the gesture in a manner that may cause the computingsystem to perform an action that is based on the detected gesture. Forexample, referring to FIG. 1, the gesture response identifier 110 mayadjust a threshold for determining whether to trigger the response tothe gesture. The adjustment to the threshold may be based at least inpart on the context in which the gesture was performed. The triggeringof the response by the gesture response identifier 110 may cause theaction executer 112 to perform an action on the receiving device 104based on the detected gesture.

The systems described herein may perform step 830 in a variety of ways.In one example, the estimator 606 may provide an estimate of acombination of one or more of the criteria 608 a-f to the gestureresponse model 610. The gesture response model 610 may receive therecognizer score 612, and an estimate of a combination of one or more ofthe criteria 608 a-f from the estimator 606. The gesture response model610 may use the estimate of the combination of the one or more of thecriteria 608 a-f provided by the estimator 606 to further refine, adapt,or update the recognizer score 612 to generate the gesture recognitionthreshold confidence level 504, which may be the adjusted threshold forresponding to the gesture. The gesture recognizer 604 may use thegesture recognition threshold confidence level 504 to determiningwhether to trigger a response to the gesture. The trigger of theresponse by the gesture recognizer 604 may cause the action executer 112to perform an action on the receiving device 104 based on the detectedgesture.

EXAMPLE EMBODIMENTS

Example 1: A computer-implemented method may include detecting, by acomputing system, a gesture that appears to be intended to trigger aresponse by the computing system, identifying, by the computing system,a context in which the gesture was performed, and adjusting, based atleast on the context in which the gesture was performed, a threshold fordetermining whether to trigger the response to the gesture in a mannerthat causes the computing system to perform an action that is based onthe detected gesture.

Example 2: The computer-implemented method of Example 1, where thecontext in which the gesture was performed may include an estimated timeinvolved in recovering from performing the action, and adjusting thethreshold for determining whether to trigger the response to the gesturemay be based on the estimated time involved in recovering fromperforming the action if the gesture was not intended to trigger theresponse and the action was performed.

Example 3: The computer-implemented method of Example 1, where thecontext in which the gesture was performed may include an estimated timeinvolved in recovering from not performing the action, and adjusting athreshold for determining whether to trigger the response to the gesturemay be based on an estimated time involved in recovering from notperforming the action if the gesture was intended to trigger theresponse and the action was not performed.

Example 4: The computer-implemented method of any of Examples 1-3,further including calculating a recognizer score that indicates aclarity of the gesture via a probability that the computing system hasaccurately interpreted the gesture, where the context in which thegesture was performed may include the clarity of the gesture, andadjusting the threshold for determining whether to trigger the responseto the gesture may be based on the probability that the gesture isintended to trigger the response.

Example 5: The computer-implemented method of any of Examples 1-4,further comprising determining whether the user who performed thegesture is involved in multitasking on the computing system, where thecontext in which the gesture was performed may include a degree of themultitasking occurring when the gesture is detected, and adjusting thethreshold for determining whether to trigger the response to the gesturemay be based on the degree of the multitasking occurring when thegesture is detected by the computing system.

Example 6: The computer-implemented method of Example-5, where thedegree of multitasking may be based on at least one of a number ofapplications running on the computing system or a number of activitiesbeing performed by a user of the computing system.

Example 7: The computer-implemented method of any of Examples 1-6, wherethe context in which the gesture was performed may include an amount oftime involved in detecting the gesture, and adjusting the threshold fordetermining whether to trigger the response to the gesture may be basedon the amount of time involved in detecting the gesture.

Example 8: The computer-implemented method of Example 7, where theamount of time involved in detecting the gesture may be based on atleast one of an application running on the computing system, an activitybeing performed by a user of the computing system, or a speed of abehavior of a user of the computing system.

Example 9: The computer-implemented method of any of Examples 1-8, wheredetecting the gesture may include detecting one of a sequence ofgestures intended to trigger the response by the computing system, andperforming the action based on the detected gesture may includeperforming the action based on the sequence of gestures.

Example 10: The computer-implemented method of any of Examples 1-9,where the context in which the gesture was performed may include one ormore of a time-of-day, a level of ambient lighting, a state of thecomputing system, or a location where the gesture was performed.

Example 11: A system may include at least one physical processor, andphysical memory including computer-executable instructions that, whenexecuted by the physical processor, cause the physical processor todetect, by a computing device, a gesture that appears to be intended totrigger a response by the system, identify, by the computing device, acontext in which the gesture was performed, and adjust, based at leaston the context in which the gesture was performed, a threshold fordetermining whether to trigger the response to the gesture in a mannerthat causes the system to perform an action that is based on thedetected gesture.

Example 12: The system of Example 11, where the context in which thegesture was performed may include an estimated time involved inrecovering from performing the action, and adjusting the threshold fordetermining whether to trigger the response to the gesture may be basedon the estimated time involved in recovering from performing the actionif the gesture was not intended to trigger the response and the actionwas performed.

Example 13: The system of Example 11, where the context in which thegesture was performed may include an estimated time involved inrecovering from not performing the action, and adjusting a threshold fordetermining whether to trigger the response to the gesture may be basedon an estimated time involved in recovering from not performing theaction if the gesture was intended to trigger the response and theaction was not performed.

Example 14: The system of any of Examples 11-13, further includingcomputer-executable instructions that, when executed by the physicalprocessor, cause the physical processor to calculate a recognizer scorethat indicates a clarity of the gesture via a probability that thesystem has accurately interpreted the gesture, where the context inwhich the gesture was performed may include the clarity of the gesture,and adjusting the threshold for determining whether to trigger theresponse to the gesture may be based on the probability that the gestureis intended to trigger the response.

Example 15: The system of any of Examples 11-14, further includingcomputer-executable instructions that, when executed by the physicalprocessor, cause the physical processor to determine whether the userwho performed the gesture is involved in multitasking on the system,where the context in which the gesture was performed may include adegree of the multitasking occurring when the gesture is detected, andadjusting the threshold for determining whether to trigger the responseto the gesture may be based on the degree of the multitasking occurringwhen the gesture is detected by the system.

Example 16: The system of Example 15, where the degree of multitaskingmay be based on at least one of a number of applications running on thesystem or a number of activities being performed by a user of thesystem.

Example 17: The system of any of Examples 11-16, where the context inwhich the gesture was performed may include an amount of time involvedin detecting the gesture, and adjusting the threshold for determiningwhether to trigger the response to the gesture may be based on theamount of time involved in detecting the gesture.

Example 18: The system of Example 17, where the amount of time involvedin detecting the gesture may be based on at least one of an applicationrunning on the system, an activity being performed by a user of thesystem, or a speed of a behavior of a user of the system.

Example 19: The system of any of Examples 11-18, where detecting thegesture may include detecting one of a sequence of gestures intended totrigger the response by the system, and performing the action based onthe detected gesture comprises performing the action based on thesequence of gestures.

Example 20: A non-transitory computer-readable medium including one ormore computer-executable instructions that, when executed by at leastone processor of a computing device of a computing system, may cause thecomputing device to detect a gesture that appears to be intended totrigger a response by the computing system, identify, by the computingdevice, a context in which the gesture was performed, and adjust, basedat least on the context in which the gesture was performed, a thresholdfor determining whether to trigger the response to the gesture in amanner that causes the computing system to perform an action that isbased on the detected gesture.

FIG. 9 is a block diagram of an example system 900 that includes modulesfor use in adaptive input thresholding based on the temporal costs oferror correction and/or user tasks. Modules 920 may include the gesturereceiver 106, the gesture detector 108, the gesture response identifier110, and the action executer 112. Although illustrated as separateelements, one or more of modules 920 in FIG. 9 may represent portions ofa single module or application.

In certain embodiments, one or more of modules 920 in FIG. 9 mayrepresent one or more software applications or programs that, whenexecuted by a computing device, may cause the computing device toperform one or more tasks. As illustrated in FIG. 9, example system 900may also include one or more memory devices, such as memory 910. Memory910 generally represents any type or form of volatile or non-volatilestorage device or medium capable of storing data and/orcomputer-readable instructions. In one example, memory 910 may store,load, and/or maintain one or more of modules DD20. Examples of memory910 include, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives(SSDs), optical disk drives, caches, variations or combinations of oneor more of the same, and/or any other suitable storage memory.

As illustrated in FIG. 9, example system 900 may also include one ormore physical processors, such as physical processor 930. Physicalprocessor 930 generally represents any type or form ofhardware-implemented processing unit capable of interpreting and/orexecuting computer-readable instructions. In one example, physicalprocessor 930 may access and/or modify one or more of modules 920 storedin memory 910. Additionally, or alternatively, physical processor 930may execute one or more of modules 920. Examples of physical processorDD30 include, without limitation, microprocessors, microcontrollers,Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs)that implement softcore processors, Application-Specific IntegratedCircuits (ASICs), portions of one or more of the same, variations orcombinations of one or more of the same, and/or any other suitablephysical processor.

As illustrated in FIG. 9, example system 900 may also include one ormore additional elements 940. The additional elements 940 generallyrepresent any type or form of hardware and/or software. In one example,physical processor 930 may access and/or modify one or more of theadditional elements 940.

One or more repositories may include the additional elements 940. Theone or more repositories may be memory (e.g., the memory 910). The oneor more repositories may be databases. In some implementations, theadditional elements 940 may be included (part of) the system 900. Insome implementations, the additional elements 940 may be external to thesystem 900 and accessible by the system 900. The additional elements 940may include the gesture source 102.

FIG. 10 illustrates an exemplary network environment 1000 in whichaspects of the present disclosure may be implemented. The networkenvironment 1000 may include one or more computing devices (e.g.,receiving computing device 1002 and input computing device 1006) and anetwork 1004. In one example, referring to FIG. 1, the input computingdevice 1006 may be a gesture source (e.g., the gesture source 102) andthe receiving computing device 1002 may be a computing device thatreceived gesture information and data from a gesture source (e.g., thereceiving device 104).

In this example, the input computing device 1006 may include a physicalprocessor 1070 that may be one or more general-purpose processors thatexecute software instructions. The input computing device 1006 mayinclude a data storage subsystem that includes a memory 1080 which maystore software instructions, along with data (e.g., input and/or outputdata) processed by execution of those instructions. The memory 1080 mayinclude modules 1090 that may be used to control the operation of theinput computing device 1006. The input computing device 1006 may includeadditional elements 1060. In some implementations, all or part of theadditional elements 1060 may be external to the input computing device1006 and the receiving computing device 1002 and may be accessible bythe input computing device 1006 either directly (a direct connection) orby way of the network 1004.

The receiving computing device 1002 may represent a client device or auser device, such a desktop computer, laptop computer, tablet device,smartphone, or other computing device. In some implementations, thereceiving computing device 1002 may be part of or included in augmentedreality glasses, virtual reality headsets, virtual-reality environments,and/or augmented -reality environments, examples of which are describedherein with reference to FIGS. 11-15.

Referring to FIG. 9, the receiving computing device 1002 may include aphysical processor (e.g., physical processor 930), which may represent asingle processor or multiple processors, and one or more memory devices(e.g., memory 910), which may store instructions (e.g., softwareapplications) and/or data in one or more modules 920. The modules 920may store software instructions, along with data (e.g., input and/oroutput data) processed by execution of those instructions. The receivingcomputing device 1002 may include additional elements 1020. Referring toFIG. 9, the additional elements 1020 may include all or part of theadditional elements 940.

The receiving computing device 1002 may be communicatively coupled tothe input computing device 1006 through the network 1004. The network1004 may be any communication network, such as the Internet, a Wide AreaNetwork (WAN), or a Local Area Network (LAN), and may include varioustypes of communication protocols and physical connections. The inputcomputing device 1006 may communicatively connect to and/or interfacewith various devices through the network 1004. In some embodiments, thenetwork 1004 may support communication protocols such as transmissioncontrol protocol/Internet protocol (TCP/IP), Internet packet exchange(IPX), systems network architecture (SNA), and/or any other suitablenetwork protocols. In some embodiments, data may be transmitted by thenetwork 1004 using a mobile network (such as a mobile telephone network,cellular network, satellite network, or other mobile network), a publicswitched telephone network (PSTN), wired communication protocols (e.g.,Universal Serial Bus (USB), Controller Area Network (CAN)), and/orwireless communication protocols (e.g., wireless LAN (WLAN) technologiesimplementing the IEEE 802.11 family of standards, Bluetooth, BluetoothLow Energy, Near Field Communication (NFC), Z-Wave, and ZigBee).

Embodiments of the present disclosure may include or be implemented inconjunction with various types of artificial-reality systems. Artificialreality is a form of reality that has been adjusted in some mannerbefore presentation to a user, which may include, for example, a virtualreality, an augmented reality, a mixed reality, a hybrid reality, orsome combination and/or derivative thereof. Artificial-reality contentmay include completely computer-generated content or computer-generatedcontent combined with captured (e.g., real-world) content. Theartificial-reality content may include video, audio, haptic feedback, orsome combination thereof, any of which may be presented in a singlechannel or in multiple channels (such as stereo video that produces athree-dimensional (3D) effect to the viewer). Additionally, in someembodiments, artificial reality may also be associated withapplications, products, accessories, services, or some combinationthereof, that are used to, for example, create content in an artificialreality and/or are otherwise used in (e.g., to perform activities in) anartificial reality.

Artificial-reality systems may be implemented in a variety of differentform factors and configurations. Some artificial-reality systems may bedesigned to work without near-eye displays (NEDs). Otherartificial-reality systems may include an NED that also providesvisibility into the real world (such as, e.g., augmented-reality system1100 in FIG. 11) or that visually immerses a user in an artificialreality (such as, e.g., virtual-reality system 1200 in FIG. 12). Whilesome artificial-reality devices may be self-contained systems, otherartificial-reality devices may communicate and/or coordinate withexternal devices to provide an artificial-reality experience to a user.Examples of such external devices include handheld controllers, mobiledevices, desktop computers, devices worn by a user, devices worn by oneor more other users, and/or any other suitable external system.

Turning to FIG. 11, augmented-reality system 1100 may include an eyeweardevice 1102 with a frame 1110 configured to hold a left display device1115(A) and a right display device 1115(B) in front of a user's eyes.Display devices 1115(A) and 1115(B) may act together or independently topresent an image or series of images to a user. While augmented-realitysystem 1100 includes two displays, embodiments of this disclosure may beimplemented in augmented-reality systems with a single NED or more thantwo NEDs.

In some embodiments, augmented-reality system 1100 may include one ormore sensors, such as sensor 1140. Sensor 1140 may generate measurementsignals in response to motion of augmented-reality system 1100 and maybe located on substantially any portion of frame 1110. Sensor 1140 mayrepresent one or more of a variety of different sensing mechanisms, suchas a position sensor, an inertial measurement unit (IMU), a depth cameraassembly, a structured light emitter and/or detector, or any combinationthereof. In some embodiments, augmented-reality system 1100 may or maynot include sensor 1140 or may include more than one sensor. Inembodiments in which sensor 1140 includes an IMU, the IMU may generatecalibration data based on measurement signals from sensor 1140. Examplesof sensor 1140 may include, without limitation, accelerometers,gyroscopes, magnetometers, other suitable types of sensors that detectmotion, sensors used for error correction of the IMU, or somecombination thereof.

In some examples, augmented-reality system 1100 may also include amicrophone array with a plurality of acoustic transducers1120(A)-1120(J), referred to collectively as acoustic transducers 1120.Acoustic transducers 1120 may represent transducers that detect airpressure variations induced by sound waves. Each acoustic transducer1120 may be configured to detect sound and convert the detected soundinto an electronic format (e.g., an analog or digital format). Themicrophone array in FIG. 11 may include, for example, ten acoustictransducers: 1120(A) and 1120(B), which may be designed to be placedinside a corresponding ear of the user, acoustic transducers 1120(C),1120(D), 1120(E), 1120(F), 1120(G), and 1120(H), which may be positionedat various locations on frame 1110, and/or acoustic transducers 1120(I)and 1120(J), which may be positioned on a corresponding neckband 1105.

In some embodiments, one or more of acoustic transducers 1120(A)-(J) maybe used as output transducers (e.g., speakers). For example, acoustictransducers 1120(A) and/or 1120(B) may be earbuds or any other suitabletype of headphone or speaker.

The configuration of acoustic transducers 1120 of the microphone arraymay vary. While augmented-reality system 1100 is shown in FIG. 11 ashaving ten acoustic transducers 1120, the number of acoustic transducers1120 may be greater or less than ten. In some embodiments, using highernumbers of acoustic transducers 1120 may increase the amount of audioinformation collected and/or the sensitivity and accuracy of the audioinformation. In contrast, using a lower number of acoustic transducers1120 may decrease the computing power required by an associatedcontroller 1150 to process the collected audio information. In addition,the position of each acoustic transducer 1120 of the microphone arraymay vary. For example, the position of an acoustic transducer 1120 mayinclude a defined position on the user, a defined coordinate on frame1110, an orientation associated with each acoustic transducer 1120, orsome combination thereof.

Acoustic transducers 1120(A) and 1120(B) may be positioned on differentparts of the user's ear, such as behind the pinna, behind the tragus,and/or within the auricle or fossa. Or, there may be additional acoustictransducers 1120 on or surrounding the ear in addition to acoustictransducers 1120 inside the ear canal. Having an acoustic transducer1120 positioned next to an ear canal of a user may enable the microphonearray to collect information on how sounds arrive at the ear canal. Bypositioning at least two of acoustic transducers 1120 on either side ofa user's head (e.g., as binaural microphones), augmented-reality device1100 may simulate binaural hearing and capture a 3D stereo sound fieldaround about a user's head. In some embodiments, acoustic transducers1120(A) and 1120(B) may be connected to augmented-reality system 1100via a wired connection 1130, and in other embodiments acoustictransducers 1120(A) and 1120(B) may be connected to augmented-realitysystem 1100 via a wireless connection (e.g., a BLUETOOTH connection). Instill other embodiments, acoustic transducers 1120(A) and 1120(B) maynot be used at all in conjunction with augmented-reality system 1100.

Acoustic transducers 1120 on frame 1110 may be positioned in a varietyof different ways, including along the length of the temples, across thebridge, above or below display devices 1115(A) and 1115(B), or somecombination thereof. Acoustic transducers 1120 may also be oriented suchthat the microphone array is able to detect sounds in a wide range ofdirections surrounding the user wearing the augmented-reality system1100. In some embodiments, an optimization process may be performedduring manufacturing of augmented-reality system 1100 to determinerelative positioning of each acoustic transducer 1120 in the microphonearray.

In some examples, augmented-reality system 1100 may include or beconnected to an external device (e.g., a paired device), such asneckband 1105. Neckband 1105 generally represents any type or form ofpaired device. Thus, the following discussion of neckband 1105 may alsoapply to various other paired devices, such as charging cases, smartwatches, smart phones, wrist bands, other wearable devices, hand-heldcontrollers, tablet computers, laptop computers, other external computedevices, etc.

As shown, neckband 1105 may be coupled to eyewear device 1102 via one ormore connectors. The connectors may be wired or wireless and may includeelectrical and/or non-electrical (e.g., structural) components. In somecases, eyewear device 1102 and neckband 1105 may operate independentlywithout any wired or wireless connection between them. While FIG. 11illustrates the components of eyewear device 1102 and neckband 1105 inexample locations on eyewear device 1102 and neckband 1105, thecomponents may be located elsewhere and/or distributed differently oneyewear device 1102 and/or neckband 1105. In some embodiments, thecomponents of eyewear device 1102 and neckband 1105 may be located onone or more additional peripheral devices paired with eyewear device1102, neckband 1105, or some combination thereof.

Pairing external devices, such as neckband 1105, with augmented-realityeyewear devices may enable the eyewear devices to achieve the formfactor of a pair of glasses while still providing sufficient battery andcomputation power for expanded capabilities. Some or all of the batterypower, computational resources, and/or additional features ofaugmented-reality system 1100 may be provided by a paired device orshared between a paired device and an eyewear device, thus reducing theweight, heat profile, and form factor of the eyewear device overallwhile still retaining desired functionality. For example, neckband 1105may allow components that would otherwise be included on an eyeweardevice to be included in neckband 1105 since users may tolerate aheavier weight load on their shoulders than they would tolerate on theirheads. Neckband 1105 may also have a larger surface area over which todiffuse and disperse heat to the ambient environment. Thus, neckband1105 may allow for greater battery and computation capacity than mightotherwise have been possible on a stand-alone eyewear device. Sinceweight carried in neckband 1105 may be less invasive to a user thanweight carried in eyewear device 1102, a user may tolerate wearing alighter eyewear device and carrying or wearing the paired device forgreater lengths of time than a user would tolerate wearing a heavystandalone eyewear device, thereby enabling users to more fullyincorporate artificial-reality environments into their day-to-dayactivities.

Neckband 1105 may be communicatively coupled with eyewear device 1102and/or to other devices. These other devices may provide certainfunctions (e.g., tracking, localizing, depth mapping, processing,storage, etc.) to augmented-reality system 1100. In the embodiment ofFIG. 11, neckband 1105 may include two acoustic transducers (e.g.,1120(I) and 1120(J)) that are part of the microphone array (orpotentially form their own microphone subarray). Neckband 1105 may alsoinclude a controller 1125 and a power source 1135.

Acoustic transducers 1120(I) and 1120(J) of neckband 1105 may beconfigured to detect sound and convert the detected sound into anelectronic format (analog or digital). In the embodiment of FIG. 11,acoustic transducers 1120(I) and 1120(J) may be positioned on neckband1105, thereby increasing the distance between the neckband acoustictransducers 1120(I) and 1120(J) and other acoustic transducers 1120positioned on eyewear device 1102. In some cases, increasing thedistance between acoustic transducers 1120 of the microphone array mayimprove the accuracy of beamforming performed via the microphone array.For example, if a sound is detected by acoustic transducers 1120(C) and1120(D) and the distance between acoustic transducers 1120(C) and1120(D) is greater than, e.g., the distance between acoustic transducers1120(D) and 1120(E), the determined source location of the detectedsound may be more accurate than if the sound had been detected byacoustic transducers 1120(D) and 1120(E).

Controller 1125 of neckband 1105 may process information generated bythe sensors on neckband 1105 and/or augmented-reality system 1100. Forexample, controller 1125 may process information from the microphonearray that describes sounds detected by the microphone array. For eachdetected sound, controller 1125 may perform a direction-of-arrival (DOA)estimation to estimate a direction from which the detected sound arrivedat the microphone array. As the microphone array detects sounds,controller 1125 may populate an audio data set with the information. Inembodiments in which augmented-reality system 1100 includes an inertialmeasurement unit, controller 1125 may compute all inertial and spatialcalculations from the IMU located on eyewear device 1102. A connectormay convey information between augmented-reality system 1100 andneckband 1105 and between augmented-reality system 1100 and controller1125. The information may be in the form of optical data, electricaldata, wireless data, or any other transmittable data form. Moving theprocessing of information generated by augmented-reality system 1100 toneckband 1105 may reduce weight and heat in eyewear device 1102, makingit more comfortable to the user.

Power source 1135 in neckband 1105 may provide power to eyewear device1102 and/or to neckband 1105. Power source 1135 may include, withoutlimitation, lithium ion batteries, lithium-polymer batteries, primarylithium batteries, alkaline batteries, or any other form of powerstorage. In some cases, power source 1135 may be a wired power source.Including power source 1135 on neckband 1105 instead of on eyeweardevice 1102 may help better distribute the weight and heat generated bypower source 1135.

As noted, some artificial-reality systems may, instead of blending anartificial reality with actual reality, substantially replace one ormore of a user's sensory perceptions of the real world with a virtualexperience. One example of this type of system is a head-worn displaysystem, such as virtual-reality system 1200 in FIG. 12, that mostly orcompletely covers a user's field of view. Virtual-reality system 1200may include a front rigid body 1202 and a band 1204 shaped to fit arounda user's head. Virtual-reality system 1200 may also include output audiotransducers 1206(A) and 1206(B). Furthermore, while not shown in FIG.12, front rigid body 1202 may include one or more electronic elements,including one or more electronic displays, one or more inertialmeasurement units (IMUS), one or more tracking emitters or detectors,and/or any other suitable device or system for creating anartificial-reality experience.

Artificial-reality systems may include a variety of types of visualfeedback mechanisms. For example, display devices in augmented-realitysystem 1100 and/or virtual-reality system 1200 may include one or moreliquid crystal displays (LCDs), light emitting diode (LED) displays,microLED displays, organic LED (OLED) displays, digital light project(DLP) micro-displays, liquid crystal on silicon (LCoS) micro-displays,and/or any other suitable type of display screen. Theseartificial-reality systems may include a single display screen for botheyes or may provide a display screen for each eye, which may allow foradditional flexibility for varifocal adjustments or for correcting auser's refractive error. Some of these artificial-reality systems mayalso include optical subsystems having one or more lenses (e.g.,conventional concave or convex lenses, Fresnel lenses, adjustable liquidlenses, etc.) through which a user may view a display screen. Theseoptical subsystems may serve a variety of purposes, including tocollimate (e.g., make an object appear at a greater distance than itsphysical distance), to magnify (e.g., make an object appear larger thanits actual size), and/or to relay (to, e.g., the viewer's eyes) light.These optical subsystems may be used in a non-pupil-forming architecture(such as a single lens configuration that directly collimates light butresults in so-called pincushion distortion) and/or a pupil-formingarchitecture (such as a multi-lens configuration that produces so-calledbarrel distortion to nullify pincushion distortion).

In addition to or instead of using display screens, some of theartificial-reality systems described herein may include one or moreprojection systems. For example, display devices in augmented-realitysystem 1100 and/or virtual-reality system 1200 may include micro-LEDprojectors that project light (using, e.g., a waveguide) into displaydevices, such as clear combiner lenses that allow ambient light to passthrough. The display devices may refract the projected light toward auser's pupil and may enable a user to simultaneously view bothartificial-reality content and the real world. The display devices mayaccomplish this using any of a variety of different optical components,including waveguide components (e.g., holographic, planar, diffractive,polarized, and/or reflective waveguide elements), light-manipulationsurfaces and elements (such as diffractive, reflective, and refractiveelements and gratings), coupling elements, etc. Artificial-realitysystems may also be configured with any other suitable type or form ofimage projection system, such as retinal projectors used in virtualretina displays.

The artificial-reality systems described herein may also include varioustypes of computer vision components and subsystems. For example,augmented-reality system 1100 and/or virtual-reality system 1200 mayinclude one or more optical sensors, such as two-dimensional (2D) or 3Dcameras, structured light transmitters and detectors, time-of-flightdepth sensors, single-beam or sweeping laser rangefinders, 3D LiDARsensors, and/or any other suitable type or form of optical sensor. Anartificial-reality system may process data from one or more of thesesensors to identify a location of a user, to map the real world, toprovide a user with context about real-world surroundings, and/or toperform a variety of other functions.

The artificial-reality systems described herein may also include one ormore input and/or output audio transducers. Output audio transducers mayinclude voice coil speakers, ribbon speakers, electrostatic speakers,piezoelectric speakers, bone conduction transducers, cartilageconduction transducers, tragus-vibration transducers, and/or any othersuitable type or form of audio transducer. Similarly, input audiotransducers may include condenser microphones, dynamic microphones,ribbon microphones, and/or any other type or form of input transducer.In some embodiments, a single transducer may be used for both audioinput and audio output.

In some embodiments, the artificial-reality systems described herein mayalso include tactile (i.e., haptic) feedback systems, which may beincorporated into headwear, gloves, body suits, handheld controllers,environmental devices (e.g., chairs, floormats, etc.), and/or any othertype of device or system. Haptic feedback systems may provide varioustypes of cutaneous feedback, including vibration, force, traction,texture, and/or temperature. Haptic feedback systems may also providevarious types of kinesthetic feedback, such as motion and compliance.Haptic feedback may be implemented using motors, piezoelectricactuators, fluidic systems, and/or a variety of other types of feedbackmechanisms. Haptic feedback systems may be implemented independent ofother artificial-reality devices, within other artificial-realitydevices, and/or in conjunction with other artificial-reality devices.

By providing haptic sensations, audible content, and/or visual content,artificial-reality systems may create an entire virtual experience orenhance a user's real-world experience in a variety of contexts andenvironments. For instance, artificial-reality systems may assist orextend a user's perception, memory, or cognition within a particularenvironment. Some systems may enhance a user's interactions with otherpeople in the real world or may enable more immersive interactions withother people in a virtual world. Artificial-reality systems may also beused for educational purposes (e.g., for teaching or training inschools, hospitals, government organizations, military organizations,business enterprises, etc.), entertainment purposes (e.g., for playingvideo games, listening to music, watching video content, etc.), and/orfor accessibility purposes (e.g., as hearing aids, visual aids, etc.).The embodiments disclosed herein may enable or enhance a user'sartificial-reality experience in one or more of these contexts andenvironments and/or in other contexts and environments.

As noted, artificial-reality systems 1100 and 1200 may be used with avariety of other types of devices to provide a more compellingartificial-reality experience. These devices may be haptic interfaceswith transducers that provide haptic feedback and/or that collect hapticinformation about a user's interaction with an environment. Theartificial-reality systems disclosed herein may include various types ofhaptic interfaces that detect or convey various types of hapticinformation, including tactile feedback (e.g., feedback that a userdetects via nerves in the skin, which may also be referred to ascutaneous feedback) and/or kinesthetic feedback (e.g., feedback that auser detects via receptors located in muscles, joints, and/or tendons).

Haptic feedback may be provided by interfaces positioned within a user'senvironment (e.g., chairs, tables, floors, etc.) and/or interfaces onarticles that may be worn or carried by a user (e.g., gloves,wristbands, etc.). As an example, FIG. 13 illustrates a vibrotactilesystem 1300 in the form of a wearable glove (haptic device 1310) andwristband (haptic device 1320). Haptic device 1310 and haptic device1320 are shown as examples of wearable devices that include a flexible,wearable textile material 1330 that is shaped and configured forpositioning against a user's hand and wrist, respectively. Thisdisclosure also includes vibrotactile systems that may be shaped andconfigured for positioning against other human body parts, such as afinger, an arm, a head, a torso, a foot, or a leg. By way of example andnot limitation, vibrotactile systems according to various embodiments ofthe present disclosure may also be in the form of a glove, a headband,an armband, a sleeve, a head covering, a sock, a shirt, or pants, amongother possibilities. In some examples, the term “textile” may includeany flexible, wearable material, including woven fabric, non-wovenfabric, leather, cloth, a flexible polymer material, compositematerials, etc.

One or more vibrotactile devices 1340 may be positioned at leastpartially within one or more corresponding pockets formed in textilematerial 1330 of vibrotactile system 1300. Vibrotactile devices 1340 maybe positioned in locations to provide a vibrating sensation (e.g.,haptic feedback) to a user of vibrotactile system 1300. For example,vibrotactile devices 1340 may be positioned against the user'sfinger(s), thumb, or wrist, as shown in FIG. 13. Vibrotactile devices1340 may, in some examples, be sufficiently flexible to conform to orbend with the user's corresponding body part(s).

A power source 1350 (e.g., a battery) for applying a voltage to thevibrotactile devices 1340 for activation thereof may be electricallycoupled to vibrotactile devices 1340, such as via conductive wiring1352. In some examples, each of vibrotactile devices 1340 may beindependently electrically coupled to power source 1350 for individualactivation. In some embodiments, a processor 1360 may be operativelycoupled to power source 1350 and configured (e.g., programmed) tocontrol activation of vibrotactile devices 1340.

Vibrotactile system 1300 may be implemented in a variety of ways. Insome examples, vibrotactile system 1300 may be a standalone system withintegral subsystems and components for operation independent of otherdevices and systems. As another example, vibrotactile system 1300 may beconfigured for interaction with another device or system 1370. Forexample, vibrotactile system 1300 may, in some examples, include acommunications interface 1380 for receiving and/or sending signals tothe other device or system 1370. The other device or system 1370 may bea mobile device, a gaming console, an artificial-reality (e.g.,virtual-reality, augmented-reality, mixed-reality) device, a personalcomputer, a tablet computer, a network device (e.g., a modem, a router,etc.), a handheld controller, etc. Communications interface 1380 mayenable communications between vibrotactile system 1300 and the otherdevice or system 1370 via a wireless (e.g., Wi-Fi, BLUETOOTH, cellular,radio, etc.) link or a wired link. If present, communications interface1380 may be in communication with processor 1360, such as to provide asignal to processor 1360 to activate or deactivate one or more of thevibrotactile devices 1340.

Vibrotactile system 1300 may optionally include other subsystems andcomponents, such as touch-sensitive pads 1390, pressure sensors, motionsensors, position sensors, lighting elements, and/or user interfaceelements (e.g., an on/off button, a vibration control element, etc.).During use, vibrotactile devices 1340 may be configured to be activatedfor a variety of different reasons, such as in response to the user'sinteraction with user interface elements, a signal from the motion orposition sensors, a signal from the touch-sensitive pads 1390, a signalfrom the pressure sensors, a signal from the other device or system1370, etc.

Although power source 1350, processor 1360, and communications interface1380 are illustrated in FIG. 13 as being positioned in haptic device1320, the present disclosure is not so limited. For example, one or moreof power source 1350, processor 1360, or communications interface 1380may be positioned within haptic device 1310 or within another wearabletextile.

Haptic wearables, such as those shown in and described in connectionwith FIG. 13, may be implemented in a variety of types ofartificial-reality systems and environments. FIG. 14 shows an exampleartificial-reality environment 1400 including one head-mountedvirtual-reality display and two haptic devices (i.e., gloves), and inother embodiments any number and/or combination of these components andother components may be included in an artificial-reality system. Forexample, in some embodiments there may be multiple head-mounted displayseach having an associated haptic device, with each head-mounted displayand each haptic device communicating with the same console, portablecomputing device, or other computing system.

Head-mounted display 1402 generally represents any type or form ofvirtual-reality system, such as virtual-reality system 1200 in FIG. 12.Haptic device 1404 generally represents any type or form of wearabledevice, worn by a user of an artificial-reality system, that provideshaptic feedback to the user to give the user the perception that he orshe is physically engaging with a virtual object. In some embodiments,haptic device 1404 may provide haptic feedback by applying vibration,motion, and/or force to the user. For example, haptic device 1404 maylimit or augment a user's movement. To give a specific example, hapticdevice 1404 may limit a user's hand from moving forward so that the userhas the perception that his or her hand has come in physical contactwith a virtual wall. In this specific example, one or more actuatorswithin the haptic device may achieve the physical-movement restrictionby pumping fluid into an inflatable bladder of the haptic device. Insome examples, a user may also use haptic device 1404 to send actionrequests to a console. Examples of action requests include, withoutlimitation, requests to start an application and/or end the applicationand/or requests to perform a particular action within the application.

While haptic interfaces may be used with virtual-reality systems, asshown in FIG. 14, haptic interfaces may also be used withaugmented-reality systems, as shown in FIG. 15. FIG. 15 is a perspectiveview of a user 1510 interacting with an augmented-reality system 1500.In this example, user 1510 may wear a pair of augmented-reality glasses1520 that may have one or more displays 1522 and that are paired with ahaptic device 1530. In this example, haptic device 1530 may be awristband that includes a plurality of band elements 1532 and atensioning mechanism 1534 that connects band elements 1532 to oneanother.

One or more of band elements 1532 may include any type or form ofactuator suitable for providing haptic feedback. For example, one ormore of band elements 1532 may be configured to provide one or more ofvarious types of cutaneous feedback, including vibration, force,traction, texture, and/or temperature. To provide such feedback, bandelements 1532 may include one or more of various types of actuators. Inone example, each of band elements 1532 may include a vibrotactor (e.g.,a vibrotactile actuator) configured to vibrate in unison orindependently to provide one or more of various types of hapticsensations to a user. Alternatively, only a single band element or asubset of band elements may include vibrotactors.

Haptic devices 1310, 1320, 1404, and 1530 may include any suitablenumber and/or type of haptic transducer, sensor, and/or feedbackmechanism. For example, haptic devices 1310, 1320, 1404, and 1530 mayinclude one or more mechanical transducers, piezoelectric transducers,and/or fluidic transducers. Haptic devices 1310, 1320, 1404, and 1530may also include various combinations of different types and forms oftransducers that work together or independently to enhance a user'sartificial-reality experience. In one example, each of band elements1532 of haptic device 1530 may include a vibrotactor (e.g., avibrotactile actuator) configured to vibrate in unison or independentlyto provide one or more of various types of haptic sensations to a user.

FIG. 16A illustrates an exemplary human-machine interface (also referredto herein as an EMG control interface) configured to be worn around auser's lower arm or wrist as a wearable system 1600. In this example,wearable system 1600 may include sixteen neuromuscular sensors 1610(e.g., EMG sensors) arranged circumferentially around an elastic band1620 with an interior surface 1630 configured to contact a user's skin.However, any suitable number of neuromuscular sensors may be used. Thenumber and arrangement of neuromuscular sensors may depend on theparticular application for which the wearable device is used. Forexample, a wearable armband or wristband can be used to generate controlinformation for controlling an augmented reality system, a robot,controlling a vehicle, scrolling through text, controlling a virtualavatar, or any other suitable control task. As shown, the sensors may becoupled together using flexible electronics incorporated into thewireless device. FIG. 16B illustrates a cross-sectional view through oneof the sensors of the wearable device shown in FIG. 16A. In someembodiments, the output of one or more of the sensing components can beoptionally processed using hardware signal processing circuitry (e.g.,to perform amplification, filtering, and/or rectification). In otherembodiments, at least some signal processing of the output of thesensing components can be performed in software. Thus, signal processingof signals sampled by the sensors can be performed in hardware,software, or by any suitable combination of hardware and software, asaspects of the technology described herein are not limited in thisrespect. A non-limiting example of a signal processing chain used toprocess recorded data from sensors 1610 is discussed in more detailbelow with reference to FIGS. 17A and 17B.

FIGS. 17A and 17B illustrate an exemplary schematic diagram withinternal components of a wearable system with EMG sensors. As shown, thewearable system may include a wearable portion 1710 (FIG. 17A) and adongle portion 1720 (FIG. 17B) in communication with the wearableportion 1710 (e.g., via BLUETOOTH or another suitable wirelesscommunication technology). As shown in FIG. 17A, the wearable portion1710 may include skin contact electrodes 1711, examples of which aredescribed in connection with FIGS. 16A and 16B. The output of the skincontact electrodes 1711 may be provided to analog front end 1730, whichmay be configured to perform analog processing (e.g., amplification,noise reduction, filtering, etc.) on the recorded signals. The processedanalog signals may then be provided to analog-to-digital converter 1732,which may convert the analog signals to digital signals that can beprocessed by one or more computer processors. An example of a computerprocessor that may be used in accordance with some embodiments ismicrocontroller (MCU) 1734, illustrated in FIG. 17A. As shown, MCU 1734may also include inputs from other sensors (e.g., IMU sensor 1740), andpower and battery module 1742. The output of the processing performed byMCU 1734 may be provided to antenna 1750 for transmission to dongleportion 1720 shown in FIG. 17B.

Dongle portion 1720 may include antenna 1752, which may be configured tocommunicate with antenna 1750 included as part of wearable portion 1710.Communication between antennas 1750 and 1752 may occur using anysuitable wireless technology and protocol, non-limiting examples ofwhich include radiofrequency signaling and BLUETOOTH. As shown, thesignals received by antenna 1752 of dongle portion 1720 may be providedto a host computer for further processing, display, and/or for effectingcontrol of a particular physical or virtual object or objects.

Although the examples provided with reference to FIGS. 16A-16B and FIGS.17A-17B are discussed in the context of interfaces with EMG sensors, thetechniques described herein for reducing electromagnetic interferencecan also be implemented in wearable interfaces with other types ofsensors including, but not limited to, mechanomyography (MMG) sensors,sonomyography (SMG) sensors, and electrical impedance tomography (EIT)sensors. The techniques described herein for reducing electromagneticinterference can also be implemented in wearable interfaces thatcommunicate with computer hosts through wires and cables (e.g., USBcables, optical fiber cables, etc.).

As detailed above, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each include atleast one memory device and at least one physical processor.

In some examples, the term “memory device” generally refers to any typeor form of volatile or non-volatile storage device or medium capable ofstoring data and/or computer-readable instructions. In one example, amemory device may store, load, and/or maintain one or more of themodules described herein. Examples of memory devices include, withoutlimitation, Random Access Memory (RAM), Read Only Memory (ROM), flashmemory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical diskdrives, caches, variations or combinations of one or more of the same,or any other suitable storage memory.

In some examples, the term “physical processor” generally refers to anytype or form of hardware-implemented processing unit capable ofinterpreting and/or executing computer-readable instructions. In oneexample, a physical processor may access and/or modify one or moremodules stored in the above-described memory device. Examples ofphysical processors include, without limitation, microprocessors,microcontrollers, Central Processing Units (CPUs), Field-ProgrammableGate Arrays (FPGAs) that implement softcore processors,Application-Specific Integrated Circuits (ASICs), portions of one ormore of the same, variations or combinations of one or more of the same,or any other suitable physical processor.

Although illustrated as separate elements, the modules described and/orillustrated herein may represent portions of a single module orapplication. In addition, in certain embodiments one or more of thesemodules may represent one or more software applications or programsthat, when executed by a computing device, may cause the computingdevice to perform one or more tasks. For example, one or more of themodules described and/or illustrated herein may represent modules storedand configured to run on one or more of the computing devices or systemsdescribed and/or illustrated herein. One or more of these modules mayalso represent all or portions of one or more special-purpose computersconfigured to perform one or more tasks.

In addition, one or more of the modules described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, one or more of the modules recitedherein may receive data to be transformed, transform the data, output aresult of the transformation to perform an action, use the result of thetransformation to perform an action, and/or store the result of thetransformation. Additionally or alternatively, one or more of themodules recited herein may transform a processor, volatile memory,non-volatile memory, and/or any other portion of a physical computingdevice from one form to another by executing on the computing device,storing data on the computing device, and/or otherwise interacting withthe computing device.

In some embodiments, the term “computer-readable medium” generallyrefers to any form of device, carrier, or medium capable of storing orcarrying computer-readable instructions. Examples of computer-readablemedia include, without limitation, transmission-type media, such ascarrier waves, and non-transitory-type media, such as magnetic-storagemedia (e.g., hard disk drives, tape drives, and floppy disks),optical-storage media (e.g., Compact Disks (CDs), Digital Video Disks(DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-statedrives and flash media), and other distribution systems.

The process parameters and sequence of the steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various exemplary methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or include additional steps in addition to those disclosed.

The preceding description has been provided to enable others skilled inthe art to best utilize various aspects of the exemplary embodimentsdisclosed herein. This exemplary description is not intended to beexhaustive or to be limited to any precise form disclosed. Manymodifications and variations are possible without departing from thespirit and scope of the present disclosure. The embodiments disclosedherein should be considered in all respects illustrative and notrestrictive. Reference should be made to the appended claims and theirequivalents in determining the scope of the present disclosure.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of.” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and have the same meaning as the word“comprising.”

What is claimed is:
 1. A computer-implemented method comprising:detecting, by a computing system, a gesture that appears to be intendedto trigger a response by the computing system; identifying, by thecomputing system, a context in which the gesture was performed; andadjusting, based at least on the context in which the gesture wasperformed, a threshold for determining whether to trigger the responseto the gesture in a manner that causes the computing system to performan action that is based on the detected gesture.
 2. Thecomputer-implemented method of claim 1, wherein: the context in whichthe gesture was performed comprises an estimated time involved inrecovering from performing the action; and adjusting the threshold fordetermining whether to trigger the response to the gesture is based onthe estimated time involved in recovering from performing the action ifthe gesture was not intended to trigger the response and the action wasperformed.
 3. The computer-implemented method of claim 1, wherein: thecontext in which the gesture was performed comprises an estimated timeinvolved in recovering from not performing the action; and adjusting athreshold for determining whether to trigger the response to the gestureis based on an estimated time involved in recovering from not performingthe action if the gesture was intended to trigger the response and theaction was not performed.
 4. The computer-implemented method of claim 1,further comprising calculating a recognizer score that indicates aclarity of the gesture via a probability that the computing system hasaccurately interpreted the gesture, wherein: the context in which thegesture was performed comprises the clarity of the gesture; andadjusting the threshold for determining whether to trigger the responseto the gesture is based on the probability that the gesture is intendedto trigger the response.
 5. The computer-implemented method of claim 1,further comprising determining whether a user who performed the gestureis involved in multitasking on the computing system, wherein: thecontext in which the gesture was performed comprises a degree of themultitasking occurring when the gesture is detected; and adjusting thethreshold for determining whether to trigger the response to the gestureis based on the degree of the multitasking occurring when the gesture isdetected by the computing system.
 6. The computer-implemented method ofclaim 5, wherein the degree of multitasking is based on at least one ofa number of applications running on the computing system or a number ofactivities being performed by a user of the computing system.
 7. Thecomputer-implemented method of claim 1, wherein: the context in whichthe gesture was performed comprises an amount of time involved indetecting the gesture; and adjusting the threshold for determiningwhether to trigger the response to the gesture is based on the amount oftime involved in detecting the gesture.
 8. The computer-implementedmethod of claim 7, wherein the amount of time involved in detecting thegesture is based on at least one of an application running on thecomputing system, an activity being performed by a user of the computingsystem, or a speed of a behavior of a user of the computing system. 9.The computer-implemented method of claim 1, wherein: detecting thegesture comprises detecting one of a sequence of gestures intended totrigger the response by the computing system; and performing the actionbased on the detected gesture comprises performing the action based onthe sequence of gestures.
 10. The computer-implemented method of claim1, wherein the context in which the gesture was performed comprises oneor more of: a time-of-day; a level of ambient lighting; a state of thecomputing system; or a location where the gesture was performed.
 11. Asystem comprising: at least one physical processor; and physical memorycomprising computer-executable instructions that, when executed by thephysical processor, cause the physical processor to: detect, by acomputing device, a gesture that appears to be intended to trigger aresponse by the system; identify, by the computing device, a context inwhich the gesture was performed; and adjust, based at least on thecontext in which the gesture was performed, a threshold for determiningwhether to trigger the response to the gesture in a manner that causesthe system to perform an action that is based on the detected gesture.12. The system of claim 11, wherein: the context in which the gesturewas performed comprises an estimated time involved in recovering fromperforming the action; and adjusting the threshold for determiningwhether to trigger the response to the gesture is based on the estimatedtime involved in recovering from performing the action if the gesturewas not intended to trigger the response and the action was performed.13. The system of claim 11, wherein: the context in which the gesturewas performed comprises an estimated time involved in recovering fromnot performing the action; and adjusting a threshold for determiningwhether to trigger the response to the gesture is based on an estimatedtime involved in recovering from not performing the action if thegesture was intended to trigger the response and the action was notperformed.
 14. The system of claim 11, further comprisingcomputer-executable instructions that, when executed by the physicalprocessor, cause the physical processor to calculate a recognizer scorethat indicates a clarity of the gesture via a probability that thesystem has accurately interpreted the gesture, wherein: the context inwhich the gesture was performed comprises the clarity of the gesture;and adjusting the threshold for determining whether to trigger theresponse to the gesture is based on the probability that the gesture isintended to trigger the response.
 15. The system of claim 11, furthercomprising computer-executable instructions that, when executed by thephysical processor, cause the physical processor to determine whether auser who performed the gesture is involved in multitasking on thesystem, wherein: the context in which the gesture was performedcomprises a degree of the multitasking occurring when the gesture isdetected; and adjusting the threshold for determining whether to triggerthe response to the gesture is based on the degree of the multitaskingoccurring when the gesture is detected by the system.
 16. The system ofclaim 15, wherein the degree of multitasking is based on at least one ofa number of applications running on the system or a number of activitiesbeing performed by a user of the system.
 17. The system of claim 11,wherein: the context in which the gesture was performed comprises anamount of time involved in detecting the gesture; and adjusting thethreshold for determining whether to trigger the response to the gestureis based on the amount of time involved in detecting the gesture. 18.The system of claim 17, wherein the amount of time involved in detectingthe gesture is based on at least one of an application running on thesystem, an activity being performed by a user of the system, or a speedof a behavior of a user of the system.
 19. The system of claim 11,wherein: detecting the gesture comprises detecting one of a sequence ofgestures intended to trigger the response by the system; and performingthe action based on the detected gesture comprises performing the actionbased on the sequence of gestures.
 20. A non-transitorycomputer-readable medium comprising one or more computer-executableinstructions that, when executed by at least one processor of acomputing device of a computing system, cause the computing device to:detect a gesture that appears to be intended to trigger a response bythe computing system; identify, by the computing device, a context inwhich the gesture was performed; and adjust, based at least on thecontext in which the gesture was performed, a threshold for determiningwhether to trigger the response to the gesture in a manner that causesthe computing system to perform an action that is based on the detectedgesture.